OPIC HV^f CIPO 

Office de la propriety >^HQ^HHEMTJ Canadian 

e du Canada \ I Property Office 



(21) (Al) 2,129,925 

(22) 1994/08/11 
(43) 1996/02/12 



(51) Int. CI. H04N-005/278; G03B-031/00 

(19) (ca) APPLICATION FOR CANADIAN PATENT (12) 

(54) Audio Synchronization of Subtitles 



(72) Zwaneveld, Hendrik A. E. - Canada 
Dickson, Brian C. - Canada ; 
Snell, Roy C. - Canada ; 
Jaslowitz, Morris - Canada ; 



(71) Same as inventor 
(57) 10 Claims 



Notice: This application is as filed and may therefore conta 
incomplete specification. 

Industrie Canada Industry Canada 348i 



Canada 



2129925 

ABSTRACT OF THE DISCLOSURE 



A method and system (or apparatus) are provided for creating and 
presenting continuous, dialogue-matched and uninterrupted display 
5 of subtitles while displaying or presenting motion picture 

images, video images, and the like. The text of a sequence of 
subtitles to be displayed, and the in- and out-points 
corresponding to the time during which dialogue or narration 
occurs on the screen are determined by matching or comparing 
10 sound modulation samples, previously captured from the sound 
track, with sound modulation samples taken during the 
presentation. 
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Title: AUDIO SYNCHRONIZATION OF SUBTITLES 

5 The present invention relates to the display of subtitles (e.g. 
voice content) by electronic means which exploits digital audio 
signal signature sampling and digital signature pattern matching 
to synchronize or correlate the display of a subtitle with a 
particular point in time during the presentation or play back of 
10 a medium carrying an audio signal; in accordance with the present 
invention subtitles (i.e. text) may be displayed with respect to 
motion pictures, videos, slide presentations and the like or even 
with respect to audio sound tracks not associated with images. 

15 Conventional film subtitling has previously required dialogue or 
narration text to be physically incorporated into the image areas 
of a film print. The incorporation of subtitles on film may be 
accomplished for example by using known chemical or laser etching 
techniques or by known photo-optical means; thus for example, the 

20 text of a subtitle may be applied onto a latent film element 

which may be subsequently used as an overlay during the printing 
process. 

U.S patent no. 4,673,266 is an example of a subtitle presentation 
25 system which marks a film print with coded signals; each recorded 
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subtitle is recalled by means of a corresponding coded signal on 
the print and displayed. 

On the other hand, the present invention allows an unaltered 
5 presentation copy to be projected along with subtitles or 

captions which are generated and synchronized with the images by 
electronic means, i.e. under control of a micro-computer. The 
subtitles may for example appear in the image frame itself or 
alternatively on a display means located below, above or beside 
10 the image frame. 

The present invention offers an effective way to avoid the high 
costs associated with conventional subtitling processes. The 
present invention also advantageously allows a single 
15 presentation copy to be projected with subtitles in any number of 
languages or with same-language captions without having to 
permanently alter the presentation copy. 

U.S. patent nos. 4,839,744 and 5,055,939 as well as related 
20 Canadian patent no. 1,307,745 teach a system for providing a lock 
synchronization between a lower quality original (analogue) sound 
track associated with a motion picture film or the like and a 
higher quality digital sound track or source; the higher quality 
digital sound, external to the print is heard by the audience, 
25 instead of the original sound track (s) on the film print. These 
patents do not deal with the problem of displaying subtitles; 
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they deal with the replacement of one sound track by another 
sound track and thus teach away from the present invention which 
is concerned with providing text as a visual supplement to an 
actual sound track. 

5 

SUMMARY OF THE INVENTION 



Thus, in a broad aspect the present invention provides a method 
for providing subtitle co-ordination display information from a 
sound track providing an audio signal carrying an audio message, 
the display information being configured for displaying one or 
more subtitles during play of the audio message from the sound 
track, comprising 

- providing a first storage element containing subtitle cue 
information comprising one or more predetermined subtitles, 
each subtitle being associated with a respective display cue 
stamp, each display cue stamp comprising a first cue value 
and a second cue value respectively marking the beginning 
and the end of the display of a respective subtitle, 

- sensing said audio signal and providing a digital audio 
signal output to a processor element, 

said processor element being able to associate a 
digital audio signal signature (i.e. fingerprint, 
pattern or the like) with a respective predetermined 
subtitle of stage first storage element, and 
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- storing subtitle display information in a second storage 
element, said subtitle display information comprising one or 
more of the said subtitles produced by said processor 
element, each of said so stored subtitles being associated 
with a respective said audio signal signature and a 
respective said display cue stamp. 

In a further aspect the present invention provides a method for 
displaying one or more subtitles during play of an audio message 
from a sound track providing an audio signal carrying said audio 
message, comprising 

- providing a storage element containing subtitle display 
information comprising one or more predetermined subtitles 
and one or more digital audio signal signatures derived from 
said audio signal, each said subtitle being associated with 
a respective digital audio signal signature, each said 
subtitle being associated with a respective display cue 
stamp, each display cue stamp comprising a first cue value 
and a second cue value respectively marking the beginning 
and the end of the display of a respective subtitle, 

- sensing the audio signal and producing an output 
comprising a said digital audio signal signature, 

- selecting from said storage element, during play of said 
audio message, a specific subtitle by correlating the audio 
signal signatures of said output from the sensing element 
with the audio signal signatures of the storage element, and 
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- displaying the specific subtitle, the specific subtitle 
being displayed in accordance with the display cue stamp 
associated therewith. 

It is to be understood that the method can continuously sample 
the audio signal and implement the display of one subtitle after 
the other in succession in accordance with the cue data 
associated with the subtitles. 

The present invention in another broad aspect provides an audio 
cue system for providing subtitle co-ordination display 
information from a sound track providing an audio signal carrying 
an audio message, said display information being configured for 
displaying one or more subtitles during play of the audio message 
from the sound track, comprising 

- a processor element, 

- a first storage element containing subtitle cue 
information comprising one or more predetermined subtitles, 
each subtitle being associated with a respective display cue 
stamp, each display cue stamp comprising a first cue value 
and a second cue value respectively marking the beginning 
and the end of the display of a respective subtitle, 

- a sensing element for sensing said audio signal and for 
providing an input to said processor element comprising a 
digital audio signal, 
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said processor element being able to associate a 
digital audio signal signature with a respective 
predetermined subtitle of said first storage element, 
and 

- a second storage element for storing subtitle display 
information comprising one or more of the said subtitles 
produced by said processor element, each of said so stored 
subtitles being associated with a respective said audio 
signal signature and a respective said display cue stamp. 



In accordance with a particular aspect the present invention 
provides an audio cue system for providing subtitle co-ordination 
display information for displaying subtitles during projection of 
viewable images, the images having a sound track associated 
therewith for providing an audio signal carrying an audio message 
(including a first spoken language) synchronised with the 
viewable images, the display information being obtained from the 
sound track, comprising 

- a processor element, 

- a first storage element containing subtitle cue 
information comprising one or more predetermined subtitles, 
each subtitle being associated with a respective display cue 
stamp, each display cue stamp comprising a first cue value 
and a second cue value respectively marking the beginning 
and the end of the display of a respective subtitle, 
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- a sensing element for sensing said audio signal and for 
providing an input to said processor element comprising a 
digital audio signal, 

said processor element being able to associate a 
digital audio signal signature with a respective 
predetermined subtitle of said first storage element, 
and 

- a second storage element for storing subtitle display 
information comprising one or more of the said subtitles 
produced by said processor element, each of said so stored 
subtitles being associated with a respective said audio 
signal signature and a respective said display cue stamp. 

The present invention in another broad aspect provides a display 
system for displaying subtitles during play of an audio message 
from a sound track providing an audio signal carrying said audio 
message, comprising 

- a storage element containing subtitle information 
comprising one or more predetermined subtitles and one or 
more digital audio signal signatures (i.e. fingerprint, 
pattern or the like) derived from the audio signal, each 
said subtitle being associated with a respective digital 
audio signal signature, each said subtitle being associated 
with a respective display cue stamp, each display cue stamp 
comprising a first cue value and a second cue value 
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respectively marking the beginning and the end of the 
display of a respective subtitle, 

- a sensing element for sensing said audio signal and for 
producing an output comprising a said digital audio signal 
signatures, 

- a processor element for selecting from the storage 
element, during play of said audio message, a specific 
subtitle for display by correlating the audio signal 
signatures of the output from the sensing element with the 
audio signal signatures of the storage element, the 
processor element being able to produce an output 
representative of the specific subtitle, and 

- a display element for displaying the selected subtitle in 
response to the output from the processor element, the 
selected subtitle being displayed in accordance with the 
display cue stamp associated therewith. 

In a more particular aspect the present invention provides a 
display system for displaying subtitles during projection of 
viewable images, said images having a sound track associated 
therewith for providing an audio signal carrying an audio message 
(e.g. an audio message including a first spoken language) 
synchronised with said viewable images, comprising 

- a storage element containing subtitle display information 
comprising one or more predetermined subtitles and one or 
more digital audio signal signatures derived from the audio 
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signal, each said subtitle being associated with a 
respective digital audio signal signature, each said 
subtitle being associated with a respective display cue 
stamp, each display cue stamp comprising a first cue value 
and a second cue value respectively marking the beginning 
and the end of the display of a respective subtitle, 

- a sensing element for sensing the audio signal and for 
producing an output comprising a said digital audio signal 
signatures, 

- a processor element for selecting from said storage 
element, during display of said viewable images, a specific 
subtitle for display by correlating the audio signal 
signature of the output from the sensing element with the 
audio signal signatures of the storage element, the 
processor element being able to produce an output 
representative of the specific subtitle, and 

- a display element for displaying the specific subtitle in 
response to the output from the processor element, the 
selected subtitle being displayed in accordance with the 
display cue stamp associated therewith and in 
synchronization with the viewable images. 

Cue data or values as mentioned herein may be expressed in any 
suitable manner whatsoever; it may for example be expressed in 
feet, in frames, in time, in drop- or non-drop frame video mode, 
film edge keynumbers, and the like. 
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Thus, in accordance with the present invention each display cue 
stamp may for example comprise a display time stamp, each display 
time stamp comprising a first time and a second time respectively 
marking the beginning and the end of the display of a subtitle, ; 
5 in this case a selected subtitle is displayed in accordance with 
the display time stamp associated therewith. 

Alternatively, in accordance with the present invention each 
display cue stamp may if desired for example, comprise a display 
10 frame count stamp, each display frame count stamp comprising a 
first frame count value and a second frame count value 
respectively marking the beginning and the end of the display of 
a subtitle; a selected subtitle is displayed in accordance with 
the display frame count stamp associated therewith. 

15 

The text of a subtitle may be in the same language as the 
original film, video or the like or else it may be a translation 
in some other language; in either case the presentation copy of 
the film, video and the like need not be permanently altered to 
20 include the subtitle. 

As may be understood, the present invention provides a method and 
system (or apparatus) for creating and presenting continuous, 
dialogue-matched and uninterrupted display of subtitles during 
25 the presentation or play on the one hand of motion picture 

images, video images, still images and the like associated with a 
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sound track or on the other hand even of an audio message alone 
(i.e. without such images). 

Any suitable computer provided with suitable software may be used 
5 to provide the (ASCII) text of a sequence of subtitles to be 

displayed; the software should also be able to associate the in- 
and out-points corresponding to the time during which dialogue or 
narration occurs on the screen which may be recorded using any 
suitable memory medium, along with a continuous standard 
10 SMPTE/EBU time-code or frame count. 

During presentation, for example, sound modulation samples which 
may be stored in computer-based memory, are compared against 
sound modulation samples taken during the presentation, are 

15 matched and subtitles synchronized with the audio message by any 
suitable pattern matching techniques (i.e. using any suitable 
computer program) . Using commercial electronic display 
technology, the display of a subtitle text simultaneously with 
the actual dialogue or narration in the original audio on the 

20 presentation copy, may be activated by processor means (e.g. 
comprising the computer) which is prompted by the timing cue 
instructions and sound modulation samples already stored on the 
computer memory, . 

25 The basic aspects of the system relate to creating subtitle text 
and associated display cue stamps, capturing audio 
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synchronization information and associating audio signatures with 
the text and display cue stamps and finally presenting the text 
on screen on the basis of such synchronization information. The 
display system may as described herein, for example, include an 
5 electronic representation of a cuesheet, which comprises the text 
of each of a film's subtitles, along with the cuepoints 
identifying when each subtitle is required to appear and 
subsequently disappear. 

10 Initial text editing, adaptation and versioning of the cuesheet 
or text/ cue file may be done from a dialogue script, spotting 
list or ASCII file or by viewing of a timecoded video cassette, 
and may be executed from data created on commonly used text 
editing platforms, including for example, Macintosh, MS-DOS and 

15 Windows and the like. Any suitable capture software (e.g. 

spotting, translation and text editing software available from 
the National Film Board of Canada under the product name Cine- 
Text v 3.1, module 1) may be used to create an above mentioned 
initial text/ cue file. Thus, for example subtitle information 

20 comprising a subtitle and an associated display cue stamp, may be 
organized in an ASCII file that has any legal DOS filename, as 
below 

1 

25 I'm Annie Vidal 

696 
729 

2 
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I want to tell you the "Rest-O-Pop" 
story 

736 

818 

5 3 

Let me present you the "rest-O-Pop" crew. 
827 
927 

10 

In the above example file each subtitle may comprise two lines of 
text with a maximum of 40 characters per line; when only one line 
is used the other line must be left blank. In the pair of lines 
immediately following a subtitle of the cuesheet file the first 

15 line is used to present the subtitle's (first) cue-in value (i.e. 
start time) e.g. 827 seconds; the second line is used to present 
the subtitle's (second) cue-out value (i.e. finish time) e.g. 927 
seconds. In any case the cue in and cue out values are expressed 
as cumulative time or frame count values (i.e. starting at an 

20 initial time or frame count of zero) . A pair of subtitle lines 
may, if desired, be preceded by a line which is either blank or 
contains a subtitle identification number. 

The above subtitle file format respects industry conventions 
25 which call for up to 40 characters per line, with text 

positioning on the left, right or in the centre, the subtitle 
window permits the lay-out of static, non-scrolling text lines. 
Subtitles appear and disappear in their totality, as determined 
by cuepoints. Scene length determines the size of the text 
30 window and therewith the duration of a subtitle, which assures 
adequate picture-content viewing time. 
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After the text/ cue file is obtained it may, for example, then be 
used to obtain subtitle display information comprised for example 
in a file including audio signal signatures associated with 
respective subtitles. 

5 

Any suitable capture software may be used to create an above 
mentioned subtitle display information provided that it enables 
lay-out of synchronization parameters, and capture of sound track 
modulation reference samples (optical, magnetic and the like) 

10 throughout the reel, tape, compact disk and the like so as to 
enable real-time, continuous linkage or synchronization of 
picture content with subtitle/caption representations of dialogue 
or narration. Preferably, the capture software may be used in 
conjunction with standard film editing devices and is usable with 

15 standard projection equipment or video playback equipment with a 
common monitor and commonly available personal computers. A 
suitable program is the Cine-Text, v3.1 module 2, 
preparation/capturing/testing software which is available from 
the National Film Board of Canada. 

20 

A suitable computer may be equipped with a Digital Signal 
Processing (DSP) card (e.g. a Spectrum TMS320C25 system board 
card from Spectrum Signal Processing Inc. Burnaby, B.C. Canada) 
and an Audio Interface box (e.g. a CineText Audio Interface box 
25 available from the National Film Board) , which enable the system 
having suitable capture software to sample a film's soundtracks, 
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and thereby create and store digital "sound signatures" that are 
unique to that film. The computer is plugged into the sound 
reproducer for sound reference sampling/monitoring and real-time 
syncing during initial screening. 

5 

As may be surmised, the signature sampling process relates the 
behaviour of an audio signal of a film, video and the like, at 
any location, to the temporal/positional value of that location. 
Thus, during any subsequent playback of the images and/or 

10 associated soundtrack, the system or method of the present 
invention can compare the film's audio signal against the 
previously stored sound signatures to track the images (or sound 
track) in real-time. This in turn allows the system computer to 
automatically generate and correctly synchronize the electronic 

15 display of the film's subtitles/captions in accordance with its 
subtitle cuesheet file. 

Sound signature sampling may, of course, for example, be based 
upon the playback of a film's optical sound track or its final 
20 (e.g. magnetic) sound mix. In any case, the sound signatures 

should be created from a high quality reproduction of the film's 
final sound mix. The method is especially suited for subtitle 
preparation when lead-time before the screening is short, such as 
for film festivals. 

25 
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A system in accordance with the present invention may be 
significantly less expensive than alternatives, is compatible 
with low-cost personal computers and works reliably in any TV 
standard. 

5 

Any computer file containing text and sync cues as described 
herein may be stored in a computer RAM memory, on a CD-ROM disk, 
magnetic recording media (e.g. floppy or hard disk) and the like. 
Thus, data representing the sound signatures and subtitle 

10 cuesheet files for a full-length feature film, may, for example, 
be conveniently stored and transported on a single 3 1/2 inch 
floppy disk or other similar storage media. The memory media 
may, if desired, also be used for subtitling/captioning by means 
of optical, chemical or laser methods, to control text cues 

15 during the etching or printing, as well as for video data text 
display in TV and sound dubbing studios. 

Although the above discussion has been directed to associating an 
audio signal signature with a text/cue file comprising subtitles 
20 associated with display cue stamps, it is to be understood herein 
that an audio signal signature could initially be associated with 
either a subtitle or display cue stamp and thereafter the display 
cue stamp or subtitle as the case may be could thereafter be 
associated therewith. 

25 
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The present invention provides for the subtitles associated with 
suitable synchronization information to be presented or played 
back with sound film prints, video source materials, slides, 
optical or electronic overhead slides with recorded sound, and 
5 all formats of these media. A standard video data projector, 

equipped with a suitable personal computer (i.e. PC) interface, 
represents one of the most convenient types of subtitle output 
devices for this purpose. For presentation purposes however, a 
Digital Interface card is required for play back, as well as a 
10 standard commercial Patch and Motion Detector based film 

monitoring system (such as for example model FM 35 made by 
Component Engineering of Seattle, WA, USA) , if the projector is 
not already equipped with one. 

15 A suitable computer may, for example, be equipped with a Digital 
Signal Processing (DSP) card (e.g. the above mention Spectrum 
card) and an Audio Interface box (e.g. the above mentioned Audio 
Interface box from the National Film Board of Canada) , which 
enable the system having any suitable software to sample a film's 

20 soundtracks, and thereby compare "sound signatures" that are 

unique to that film. A suitable presentation software program is 
the Cine-Text, v3 . 1 module 3, presentation software which is 
available from the National Film Board of Canada. 

25 The choice of display device for electronic subtitles/captions is 
based upon consideration of theatre size and layout, since 
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different types of video projectors and electronic display 
apparatus vary in terms of their output intensities, focal 
characteristics and viewing angles which can be accommodated. 

5 In order to achieve maximum brightness and contrast, projected 
subtitles are best displayed using an auxiliary screen rather 
than the theatre's main screen. The former should be a non- 
perforated high-gain or white surface screen, sized according to 
the dimensions of the subtitles to be projected, and capable of 
10 being mounted outside (e.g. below) the theatre's normal screen. 

If light output is not a concern, and if sufficient white screen 
area is available on a theatre's normal screen (i.e. outsid) the 
area that is required for the motion picture images) , the 
electronic subtitles/captions may be projected there instead. 

15 

Standard video coax cabling between computer and video data 
projector is required. Audio modulation pattern recognition 
technology is used to reference and track the projected film in 
real-time and, thereby, to automatically generate the film's 
2 0 subtitles in closed-loop synchronization with it. 

This technique obviates the need for installing optical shaft 
encoders (or any other optical/electro-mechanical film position 
feedback devices) in a theatre, and it gives the display system 
25 the capability of automatically recalibrating itself, if and when 
necessary to accommodate missing film footage e.g. for film 
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prints that have been broken or modified and subsequently 
repaired. It can be used to track the projection of a film print 
even when the projector(s) being used suffer(s) from poor speed 
regulation. 

5 

The display system can recognize and correct automatically for 
projector or playback device speed variations. It is tolerant of 
splices which are made after capture of sound reference samples 
as well as switch-overs between projectors which are ill-timed 

10 since the system hunts for related sound patterns and if one 
cannot be found it is merely ignored. The system is thus 
projector-stop and computer-stop resistant; the system is also 
film-break or modification resistant, even, for example, when up 
to 10 feet is removed after spotting. The system or apparatus 

15 may be installed permanently or temporarily and may be configured 
to be transportable. 

Film copies are re-usable for other subtitle versions, no etching 
or laser burn-in damage occurs on the film print picture. No 
20 inherent encoding of the copy is required, no labour intensive 
attachment (and subsequent removal) of coded signal (barcoded) 
stickers to the print, to trigger in- and out-cues during 
projection are used. It follows that no decoding detector to 
read those coded signals is required. 

25 
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Subtitle text which is displayed is discrete, and suitably is 
readable at a distance. Any known electronic data display 
technology is acceptable for display, such as a video data 
display, a Liquid Crystal Display or a Light Emitting Diode (LED) 
5 display. Optionally modular, transportable LED display panels 

may be used which may be mounted on adjustable height tripods, as 
required. 

No additional operator is required in the projection booth, since 
10 the system may be run semi-automatically and only requires being 
turned on and a cursor to be placed at the film title displayed 
on the computer screen, before screening of the film. 

No or only minor mechanical modifications are required to the 
15 projector. The system may operate with platter or single reel, 
single or multiple film projector or video projector 
configurations . 

In drawings which illustrate example embodiments of the 

20 invention: 

Figure 1 is a block diagram showing a text/ cue system, 
Figure 2 is a block diagram showing an audio cue system, 
Figure 3 is a block diagram showing a display system, 
Figure 3a shows an example audio signature obtained by the cue 

25 system of figure 1 from a moving picture film for 

association with a subtitle, 
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Figure 4a shows the same digital audio signature of figure 3 but 
as sensed by the display system of figure 2 during play 
back of the moving picture film; 

Figure 5 illustrates by means of block diagrams example 
presentation set-ups exploiting audio pattern 
recognition in accordance with the present invention 

Figure 6 shows an example film strip illustrating cue placement; 

Figure 7 shows a cue detectro assembly and optical sensor of a 
film Patch and Motion detector. 

Figure 1 illustrates the preliminary text/cue processing of a 
medium carrying a sound track, i.e. to obtain a desired 
preliminary subtitle cue information file stored on an 
appropriate storage medium. For the illustrated system, the 
sound track medium is shown as a moving picture film 1 by way of 
example only; the medium could of course be any magnetic, optical 
or magneto-optical medium or the like and may if desired comprise 
only a sound track and no viewable images. 



20 The text cue system as shown in figure 1 is provided with a cue 
source element 2 which comprises a start cue element (or patch) . 
The cue source element 2 facilitates the preparation by the 
processor element 3 of display cue stamps (as defined above) 
which are derived from the film 1 and which are associated with 

25 one or more subtitles. The start cue element may comprise a 
known patch and motion detector and an associated time-code 
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generator. A patch detector establishes the starting time for 
recording or capture; a motion and film presence detector senses 
the movement of fim (in case of a break the film stops and an 
alarm is triggered) ; and the time generator establishes the time 
5 references once the starting time is established by the patch. 
The cue source element 2 may, for example, also comprise a 
photoelectric cell which converts light variations to electrical 
impulses, such as is used on projectors, editing tables, or 
optical sound readers; on the other hand the cue source element 
10 may comprise a magnetic playback head or reproduce head, used to 
sense the magnetic flux from a magnetic sound recording. 

The cue source element 2 may, for example, provide a cue value 
output 4 to the processor element 3 in the form of cumulative 

15 play time values indicative of the amount of time the film has 

been projected (as provided by the photoelectric cell or magnetic 
playback head) . Alternatively, the cue value output 4 may be in 
the form of cumulative frame numbers indicative of the amount of 
frames which have been shown (as provided by pulse codes from an 

2 0 optical shaft encoder mounted on the film transport - it is 

presumed that the relationship of the shaft to the amount of film 
is known - there are no codes on the film to provide such 
information) . 

25 The text/cue system of figure 1 includes a processor element 3 

which may comprise a suitable computer and an associated monitor, 
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the system being suitably configured to allow an operator to 
verify the scenes associated with any given pair of cue values. 
The computer is provided with a suitable program for allowing an 
operator to command the computer to associate a specific subtitle 
5 with a specific display cue stamp comprising a first and second 
cue value as described above. 

The subtitle may be provided by input 5 which may be provided 
directly from a keyboard entry or from some previously prepared 
10 ASCII file available to the computer; a suitable example program 
is the spotting, translation and text editing software available 
from the National Film Board of Canada under the product name 
Cine-Text v 3.1, module 1. 

15 The processor element 3 is suitably configured to take the above 
mentioned subtitles and respective associated display cue stamps 
and output it via output 6 to storage element 7 in the form of a 
text/ cue file on a suitable memory medium, the text/ cue file 
being intended to comprise a plurality of such specific 

20 subtitles; alternatively, each specific subtitle may be sent to 
an individual file on the memory medium. 

In operation, the operator provides a subtitle from the keyboard 
or from a previously prepared ASCII file and using the computer 
25 selects an appropriate pair of cue values obtained from the cue 
value source to create a display cue stamp for association 
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therewith. The pair of cue values is of course selected on the 
basis of where during the play of the film the subtitle must 
appear; this is predetermined by viewing on the monitor the 
portion of the film associated with the cue values. 

5 

Figure 2 illustrates the audio processing of the film medium 1 of 
figure 1, i.e. to obtain the desired subtitle display information 
file stored on an appropriate storage medium. As previously 
mentioned above, for the illustrated system, the sound track 
10 medium is shown as a moving picture film 1 by way of example 
only; the medium could as mentioned above of course be any 
magnetic, optical or magneto-optical medium or the like. 

The audio cue system as shown in figure 2 is provided with an 
15 audio sensor element 8 which also comprises a start cue element 
(or patch) . The audio sensor element 8 is configured to provide 
a processor element 9 with a digitised audio signal derived from 
the film 1 for the preparation of audio signal signatures to be 
associated with one or more subtitles in an above mentioned 
2 0 text/ cue file. The start cue element as previously mentioned may 
comprise a known patch and motion detector and an associated 
time-code generator; a patch detector establishes the starting 
time for recording or capture and the time generator establishes 
the time references once the starting time is established by the 
25 patch. The audio sensor element 8 may, for example, also 

comprise a photoelectric cell which converts light variations to 



25 



2mm 

electrical impulses, such as is used on projectors, editing 
tables, or optical sound readers; on the other hand the audio 
sensor element 8 may comprise a magnetic playback head or 
reproduce head, used to sense the magnetic flux from a magnetic 
5 sound recording. 

The duration of the audio pattern sampling (file) depends on the 
length of the reel but is continuous throughout - starting at a 
start cue patch in the leader of the reel which enables the 
10 system to correlate the audio pattern (file) with elapsed time. 
An industry standard Patch and motion detection film monitoring 
system ( e.g. model FM 35 from Component Enginering of Seattle, 
WA. USA) may be used. 

15 The audio sampling rate may be variable but may for example range 
around 10,000 samples per second. The intervals between frames 
of time of which there may for example be 50 per second may range 
between 2 0 ms and 50 ms. 

20 A digital audio signature (finger print, etc.) sample may for 

example be taken at some point well before the appearance of the 
image for which a subtitle is to appear (i.e. the audio signature 
is offset with respect to the image presentation) ; alternatively, 
the signature may be located just before the place of intended 

25 display of the subtitle to provide sufficient intrapolation time 
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to assure that the dialogue or narration continues to match with 
the image to be displayed. 

The audio sensing element 8 provides an output 10 to the 
5 processor element 9; the output 10 comprises a digitised audio 
signal. If the sound track comprises an analogue sound signal, 
samples of light variations, sensed by a photoelectric cell 
mounted on a film projector , editing table, or independent 
optical sound reader or the analogue sound values picked up by a 

10 magnetic head on an audio or video playback device is converted 
into digital word values or audio reference signatures in any 
known manner. If the sound track already comprises a digitised 
sound track such conversion will of course not be needed, i.e. a 
digitised signal may be immediately read by suitable means 

15 directly from the film. 

Each digital word or reference signature representative of an 
audio value, may be in binary code word of any desired or 
suitable length; it could be 8 bits long, 16 bits long, 32 bits 

2 0 long, and so on. Each word or signature represents a region of 
each audio signal range. Extremes of the signal voltage ranges 
are mapped into digital words wherein the first bit (the sign 
bit) indicates the polarity of each voltage. The next bit 
informs whether the voltage is in the upper or lower half of the 

25 voltage range. The following bit again divides this assigned 

range into two and so forth. The last or least significant bit 
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is the final location of the quantization interval, which in size 
corresponds to the voltage interval (i.e. space between voltage 
peaks) . 

The resulting digital pattern words form the core of the digital 
5 audio signal signature samples for the display cue file referred 
to below. 

Temporal sample elements are also captured as part of the digital 
audio signal pattern signatures and are interpolated with the 

10 "time elapsed since start" (as encoded or read by means of the 
time code generator/reader) cue during final presentation which 
provides the context for the digital audio signature pattern 
matching window. The time stamp data relates to the time encoded 
(i.e. timecode generator encoded time) during capture phase which 

15 at the Cue Patch at the head of each reel. There is no means by 
which the system can check all the data, due to time constraints 
dictated by real-time, uninterruptible presentation, in the case 
of a film break. Therefore a selectable audio signature pattern 
matching or search windo, usually may consist for example of 

20 between 5 seconds backwards and 15 seconds forwards from the 

point of a film break or interruption, is used to re-establish 
matching. 

Figure 3 shows and example of an audio signature (set at 2 0 ms 
25 interval) . 
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The processor element 9 may comprise a suitable computer and an 
associated monitor, the computer having a suitable program for 
allowing an operator to command the computer to associate a 
specific subtitle in storage element 7 (see above) with a 
5 specific digital audio signature; as mentioned above the subtitle 
in storage element 7 will be associated with a first and second 
cue value. A suitable program is the Cine-Text, v3 . 1 module 2, 
preparation/capturing/testing software which is available from 
the National Film Board of Canada. 

10 

The processor element 9 is configured to take the above mentioned 
specific subtitle and associated audio signatures and cue values 
and output it via output 10 to storage element 11 in the form of 
a display cue file on a suitable memory medium, the display cue 
15 file being intended to comprise a plurality of such specific 

subtitles; alternatively, each specific subtitle may be sent to 
an individual file on the memory medium. 

Audio signatures to be recognisable must include significant 
20 modulation variations of intensity or peaks, to distinguish them 
from even or non-variable signal periods (i.e. during periods of 
monotonous sound or silence) when no information is available to 
create signatures. 

25 In operation, the capture of appropriate audio signature samples 
is determined by the software not by human intervention. It 
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also correlates the particular signatures associated with each 
subtitle, as well as with the temporal (timecode) data provided. 

Figure 3 shows a subtitle display system. The display system has 
5 an audio signal sensor element 12 which may be the same as or 

different from the sensor 8 of the audio cue system in figure 2; 
the audio sensor element 12 may for example be part a sound 
reader of a motion picture film projector or a video playback 
device which is being used to display the moving viewable images. 

10 The presentation copy of the film 1 is exposed to the sensor 
element 12 so as to output a digital audio signal 13 to the 
processor element 14 which may be a computer associated if 
desired with a monitor (see above) . The processor element 14 is 
provided with a suitable presentation program for outputting 

15 requests 15 to the storage element 11 (see above) to send via 

output 16 audio signal signatures and associated subtitles to the 
processor element 14; the software presentation program is also 
suitable to compare the audio signal signatures with the audio 
signal from the sensor element to determine if there is a match 

20 (i.e. as in figures 3a and 4a) and if so to send a (trigger) 
output 17, representative of the subtitle associated with the 
matched signatures, to a suitable display element 18 (such as a 
video data projector, LCD or LED display) ; the subtitle being 
spotted by the trigger audio signature or pattern being displayed 

25 in accordance with the associated cue values. The subtitle is 
displayed on an appropriate screen element as described above. 
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While the subtitles are being displayed the film itself is of 
course being displayed in known manner. 

A suitable presentation program is the Cine-Text, v3 . 1 module 3, 
5 presentation software which is available from the National Film 
Board of Canada. 

In operation, the operator loads the theatre program on the 
diskettes or storag medium supplied, selects the film, loads the 
10 film print reel one on the projector and cues it to start in 

usual manner. The projector is then started and the patch and 
motion detector automatically triggers the subtitle display; 
reels are changed as reguired. 

15 Figure 5 shows possible presentation set-ups in accordance with 
the present invention; signals going from the dolby sound unit 
correspond to the signals used during the capture process, i.e. 
during the process of associating the audio signal signatures to 
the subtitles. 

20 

Figures 6 and 7 illustrate an example of a known film Patch and 
Motion Detector. The film Patch and Motion Detector consists of 
two elements. The first is the patch or cue detector which is of 
the Eddy Current Killed Oscillation (ECKO) proximity type. It 
25 responds to small foil cue patches on the film (e.g. commonly 
used to trigger curtain openings and closings, etc.). The 
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second element is the film Presence and Motion Detector which 
consists of two infrared light detectors, scanning the two edges 
of the film. Film presence will close the relay. The Motion 
Detector scans the perforations of the film. 
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The embodiments of the invention in which an exclusive property 
is claimed are defined as follows: 

1. A display system for displaying one or more subtitles during 
play of an audio message from a sound track providing an audio 
signal carrying said audio message, comprising 

- a storage element containing subtitle display information 
comprising one or more predetermined subtitles and one or 
more digital audio signal signatures derived from said audio 
signal, each said subtitle being associated with a 
respective digital audio signal signature, each said 
subtitle being associated with a respective display cue 
stamp, each display cue stamp comprising a first cue value 
and a second cue value respectively marking the beginning 
and the end of the display of a respective subtitle, 

- a sensing element for sensing said audio signal and for 
producing an output comprising a digital audio signal 
signatures, 

- a processor element for selecting from said storage 
element, during play of said audio message, a specific 
subtitle for display by correlating said audio signal 
signature of said output from the sensing element with said 
audio signal signatures of said storage element, said 
processor element being able to produce an output 
representative of said specific subtitle, and 



33 



2129925 



- a display element for displaying said specific subtitle in 
response to said output from said processor element, said 
specific subtitle being displayed in accordance with the 
display cue stamp associated therewith. 

2. A system as defined in claim 1 wherein each display cue 
stamp comprises a display time stamp, each display time stamp 
comprising a first time and a second time respectively marking 
the beginning and the end of the display of a subtitle, and 
wherein said selected subtitle is displayed in accordance with 
the display time stamp associated therewith. 

3. A system as defined in claim 1 wherein each display cue 
stamp comprises a display frame count stamp, each display frame 
count stamp comprising a first frame count value and a second 
frame count value respectively marking the beginning and the end 
of the display of a subtitle, and wherein said selected subtitle 
is displayed in accordance with the display frame count stamp 
associated therewith. 

4. A display system for displaying subtitles during projection 
of viewable images, said images having a sound track associated 
therewith for providing an audio signal carrying an audio message 
synchronised with said viewable images, comprising 
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- a storage element containing subtitle display information 
comprising one or more predetermined subtitles and one or 
more digital audio signal signatures derived from said audio 
signal, each said subtitle being associated with a 
respective audio signal signature, each said subtitle being 
associated with a respective display cue stamp, each display 
cue stamp comprising a first cue value and a second cue 
value respectively marking the beginning and the end of the 
display of a respective subtitle, 

- a sensing element for sensing said audio signal and for 
producing an output comprising a said digital audio signal 
signature, 

- a processor element for selecting from said storage 
element a specific subtitle for display by correlating said 
audio signal signature of said output from the sensing 
element with said audio signal signatures of said storage 
element, said processor element being able to produce an 
output representative of said specific subtitle, and 

- a display element for displaying said specific subtitle in 
response to said output from said processor element, said 
specific subtitle being displayed in accordance with the 
display cue stamp associated therewith and in 
synchronization with said viewable images. 

5. A system as defined in claim 4 wherein each display cue 
stamp comprises a display time stamp, each display time stamp 
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comprising a first time and a second time respectively marking 
the beginning and the end of the display of a subtitle, and 
wherein said selected subtitle is displayed in accordance with 
the display time stamp associated therewith. 

6. A system as defined in claim 4 wherein each display cue 
stamp comprises a display frame count stamp, each display frame 
count stamp comprising a first frame count value and a second 
frame count value respectively marking the beginning and the end 
of the display of a subtitle, and wherein said selected subtitle 
is displayed in accordance with the display frame count stamp 
associated therewith. 

7. A cue system for providing subtitle co-ordination display 
information from a sound track providing an audio signal carrying 
an audio message, said display information being configured for 
displaying one or more subtitles during play of said audio 
message from said sound track, comprising 

- a processor element, 

- a first storage element containing subtitle cue 
information comprising one or more predetermined subtitles, 
each subtitle being associated with a respective display cue 
stamp, each display cue stamp comprising a first cue value 
and a second cue value respectively marking the beginning 
and the end of the display of a respective subtitle, 
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- a sensing element for sensing said audio signal and for 
providing an input to said processor element comprising a 
digital audio signal, 

said processor element being able to associate a 
digital audio signal signature with a respective 
predetermined subtitle of said first storage element, 
and 

- a second storage element for storing subtitle display 
information comprising one or more of the said subtitles 
produced by said processor element, each of said so stored 
subtitles being associated with a respective said audio 
signal signature and a respective said display cue stamp. 

8. A cue system for providing subtitle co-ordination display 
information for displaying subtitles during projection of 
viewable images, said images having a sound track associated 
therewith for providing an audio signal carrying an audio message 
synchronised with said viewable images, said display information 
being obtained from said sound track, comprising 

- a processor element, 

- a first storage element containing subtitle cue 
information comprising one or more predetermined subtitles, 
each subtitle being associated with a respective display cue 
stamp, each display cue stamp comprising a first cue value 
and a second cue value respectively marking the beginning 
and the end of the display of a respective subtitle, 
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- a sensing element for sensing said audio signal and for 
providing an input to said processor element comprising a 
digital audio signal, 

said processor element being able to associate a 
digital audio signal signature with a respective 
predetermined subtitle of said first storage element, 
and 

- a second storage element for storing subtitle display 
information comprising one or more of the said subtitles 
produced by said processor element, each of said so stored 
subtitles being associated with a respective said audio 
signal signature and a respective said display cue stamp. 



9. A method for providing subtitle co-ordination display 
information from a sound track providing an audio signal carrying 
an audio message, said display information being configured for 
displaying one or more subtitles during play of said audio 
message from said sound track, comprising 

- providing a first storage element containing subtitle cue 
information comprising one or more predetermined subtitles, 
each subtitle being associated with a respective display cue 
stamp, each display cue stamp comprising a first cue value 
and a second cue value respectively marking the beginning 
and the end of the display of a respective subtitle, 

- sensing said audio signal and providing a digital audio 
signal output to a processor element, 
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said processor element being able to associate a 
digital audio signal signature with a respective 
predetermined subtitle of said first storage element, 
and 

- storing subtitle display information in a second storage 
element, said subtitle display information comprising one or 
more of the said subtitles produced by said processor 
element, each of said so stored subtitles being associated 
with a respective said audio signal signature and a 
respective said display cue stamp. 



10. A method for displaying one or more subtitles during play of 
an audio message from a sound track providing an audio signal 
carrying said audio message, comprising 

- providing a storage element containing subtitle display 
information comprising one or more predetermined subtitles 
and one or more digital audio signal signatures derived from 
said audio signal, each said subtitle being associated with 
a respective digital audio signal signature, each said 
subtitle being associated with a respective display cue 
stamp, each display cue stamp comprising a first cue value 
and a second cue value respectively marking the beginning 
and the end of the display of a subtitle, 

- sensing said audio signal and producing an output 
comprising a said digital audio signal signature. 
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- selecting from said storage element, during play of said 
audio message, a specific subtitle for display by 
correlating said audio signal signature of said output with 
said audio signal signatures of said storage element, and 

- displaying said specific subtitle, said specific subtitle 
being displayed in accordance with the display cue stamp 
associated therewith. 
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